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Abstract 

In  many  practical  scheduling  problems,  feasible  solu¬ 
tions  can  be  partially  ordered  according  to  differences 
between  the  temporal  objects  in  each  solution.  Often, 
these  orderings  can  be  computed  from  a  compact  value 
function  that  combines  the  local  preference  values  of 
the  temporal  objects.  However,  in  part  because  it  is 
natural  to  view  temporal  domains  as  continuous,  find¬ 
ing  complete,  preferred  solutions  to  these  problems  is  a 
challenging  optimization  task.  Previous  works  achieve 
tractability  by  making  restrictions  on  the  model  of  tem¬ 
poral  preferences,  including  limiting  representations  to 
binary  and  convex  preferences.  We  propose  an  appli¬ 
cation  of  Bucket  Elimination  (BE)  to  solve  problems 
with  piecewise  linear  constraints  on  temporal  prefer¬ 
ences  with  continuous  domains.  The  key  technical  hur¬ 
dle  is  developing  a  tractable  elimination  function  for 
such  constraints.  This  proof  of  concept  takes  a  step  to¬ 
ward  an  ability  to  solve  scheduling  problems  with  richer 
models  of  preference  than  previously  entertained.  Fur¬ 
ther,  it  provides  a  complementary  approach  to  exist¬ 
ing  techniques  for  restricted  models,  because  the  com¬ 
plexity  of  BE,  while  exponential  in  the  treewidth  of  the 
problem,  is  polynomial  in  its  size. 

Introduction 

Practical  problems  requiring  the  assignment  of  times  to 
planned  events  are  pervasive  in  the  world,  from  mundane 
applications  like  calendar  management  (Moffitt,  Peintner, 
&  Yorke-Smith  2006),  to  the  more  exotic  such  as  science 
observations  for  planetary  exploration  (Bresina,  Khatib,  & 
McGann  2006).  In  cases  such  as  these,  feasible  temporal 
assignments  can  be  ordered  on  the  basis  of  the  degree  to 
which  certain  preferences  are  adhered  to.  Such  preferences 
can  sometimes  be  expressed  as  a  simple  function  of  times, 
(e.g.,  it’s  preferable  that  an  event  start  as  early  as  possible) 
but  may  also  express  complex  relationships  (e.g.,  it’s  prefer¬ 
able  that  a  set  of  events  have  roughly  the  same  duration  ). 

There  are  numerous  efforts  in  representing  and  solving 
problems  in  decision-making  with  preferences.  Practical 
problems  are  often  challenging  because  of  they  involve 
many  preference  criteria  over  continuous  temporal  variables. 
Much  of  the  previous  work  in  optimization  of  temporal  pref¬ 
erences  has  sought  to  manage  the  potential  complexity  of 
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such  problems  by  limiting  the  expressive  power  of  the  lan¬ 
guage  for  expressing  preferences.  A  common  model,  the 
Simple  Temporal  Problem  with  Preferences  (STPP)  (Morris 
et  al.  2004)  expands  the  Simple  Temporal  Network  (Dechter, 
Meiri,  &  Pearl  1991)  by  allowing  numerical  preferences  on 
the  admissible  values  of  temporal  variables.  STPPs  com¬ 
bine  both  hard  constraints  that  must  be  satisfied  for  a  solu¬ 
tion  to  be  valid,  and  soft  constraint  that  assign  a  preference 
value  to  each  admissible  assignment.  Global  preferences  are 
obtained  by  summing  the  local  preferences  of  temporal  ob¬ 
jects.  STPPs  inherit  from  STNs  being  limited  to  unary  and 
binary  constraints.  Moreover,  all  existing  efficient  solution 
techniques  for  STPPs  assume  convex  or  semi-convex  pref¬ 
erences.  Notably,  the  most  significant  result  in  STPPs  is 
the  use  of  Linear  Programing  (LP)  to  solve  problems  with 
binary  piecewise  linear  convex  preferences  as  described  in 
(Morris  et  al.  2004). 

Despite  such  progress  in  solving  planning  and  scheduling 
problems  with  temporal  preferences,  restrictions  imposed  by 
binary,  convex  preference  functions  result  in  limitations  in 
the  expressive  power  of  the  language  of  constraints.  Con¬ 
sider  a  natural  temporal  preference  such  as  Maximize  the 
temporal  gap  ( duration )  between  A  and  B.  Graphically,  this 
preference  could  be  expressed  as  a  V-shaped  non-convex 
function  from  times  to  preference  values  that  reaches  0  (the 
least  preferred  value)  at  time  0. 

Removing  the  restriction  on  constraints  imposed  by  the 
STPP  paradigm  requires  an  alternative  approach  to  finding 
solutions.  A  natural  course  of  action  at  this  point  is  to  con¬ 
sider  general  approaches  to  constraint  optimization,  either 
based  on  conditioning  (backtracking,  branch  and  bound)  or 
Bucket  Elimination  (BE),  which  is  a  generalization  of  dy¬ 
namic  programming  (Dechter  1999;  2003).  The  interest  in 
this  paper  is  in  building  a  temporal  solver  based  on  BE  for 
a  model  of  time  in  which  constraints  are  piecewise  linear 
over  continuous  domains,  but  not  necessarily  binary  or  con¬ 
vex.  The  result  of  this  work  is  the  ability  to  solve  schedul¬ 
ing  problems  with  richer  models  of  preference  than  previ¬ 
ously  entertained.  Further,  we  provide  a  complementary  ap¬ 
proach  to  existing  techniques  for  restricted  models,  because 
the  complexity  of  BE,  while  exponential  in  the  treewidth  of 
the  problem,  is  polynomial  in  its  size.  The  main  contribution 
of  this  paper  is  the  proof  of  the  soundness  of  an  approach 
whose  competitiveness  will  be  explored  in  future  works. 
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Figure  1:  A  non-convex  function  over  — 10  <  t  <  10  that 
expresses  the  preference:  maximize  the  temporal  difference 
between  two  events. 


The  next  section  survey  previous  work  on  BE.  There  fol¬ 
lows  the  definition  of  the  class  of  continuous  temporal  prob¬ 
lem  considered.  It  is  based  on  piecewise  linear  value  func¬ 
tions  that  encode  both  hard  and  soft  constraints,  and  that  are 
not  necessarily  binary  or  convex.  We  then  show  that  BE 
can  be  applied  to  this  domain,  demonstrating  by  construc¬ 
tion  that  elimination  preserves  piecewise  linearity.  To  im¬ 
prove  the  efficiency  of  the  elimination  process,  the  notion 
of  the  ‘witness’  is  applied  to  the  elimination  function,  lead¬ 
ing  to  our  final  algorithm.  Finally,  preliminary  experimental 
results  using  the  full  BE  algorithm  are  summarized. 

Bucket  Elimination 

Bucket  Elimination  (BE)  is  a  complete  approach  to  solving 
constraint  optimization  problems  that  generalizes  dynamic 
programming  (Dechter  1999;  2003).  Intuitively,  BE  solves 
a  problem  by  a  series  of  replacements  of  variables  by  con¬ 
straints  that  summarize  the  effect  of  each  variable  on  the  best 
solution  to  the  problem. 

In  this  approach,  hard  and  soft  constraints  on  an  ordered 
set  of  variables  X  =  {x\,X2,  ■  ■  .xn}  are  represented  as 
numerical  functions,  here  called  value  functions  V,  defined 
over  a  subset  of  X  called  the  scope  of  V.  A  hard  constraint 
is  represented  by  a  value  function  that  takes  the  value  0  in 
each  state  where  it  is  satisfied,  and  — oc  elsewhere.  A  soft 
constraint  is  represented  by  a  function  taking  finite  values 
that  reflects  the  preferences  over  the  variables  in  its  scope. 
A  value  function  can  take  both  finite  and  infinite  values  and 
represent  both  soft  and  hard  constraints  at  the  same  time. 

Given  a  set  V  of  such  preference  functions,  bucket  B(i )  is 
defined  as  the  set  of  constraints  having  Xi  in  their  scope  as 
their  highest  variable,  according  to  the  ordering  of  X.  For 
a  fixed  i,  we  call  Xi  =  {x j1 ,  Xj2 , . . .  Xjk ,  Xi }  the  smallest 
subset  of  variables  such  that  the  scope  of  each  utility  in  B(i) 
is  included  in  Xi.  We  then  define  A_j  =  X,  \  {xi}  = 
{xjr ,  Xj2 , . . .  Xjk  }.  Technically,  A_j  regroups  all  parents  of 
Xi  in  the  induced  constraint  graph  (Dechter  1999). 

BE  eliminates  variables  in  reverse  order  by  creating  a  con¬ 


straint  Vi  for  each  x,  eliminated  following: 

Vi  =  Elinij 
which  is  defined  as: 

V(*ii  j  Xj2 , .  ■ .  Xjk )  .  Vi  (xjl ,  Xj2 , . .  ■  Xjk ) 


max 

Xi 


E  n 


'3 1  5  *3 2  j  ' 


veB(i) 


■  Xjk  !  Xi) 


(1) 


V.  is  added  to  the  bucket  B(jk),  where  jk  is  the  index  of  the 
highest  variable  in  A BE  terminates  when  all  the  vari¬ 
ables  are  eliminated  (the  resulting  function  Vo  is  then  a  con¬ 
stant  function  returning  the  highest  value  of  any  solution  to 
the  problem).  The  optimal  state  x*  that  yielded  the  highest 
value  can  be  recovered  easily  based  on  this  computation. 

The  complexity  of  BE  strongly  contrasts  with  that  of  stan¬ 
dard  conditioning  algorithm  such  as  backtracking  search  and 
branch  and  bound  (Dechter  1999).  The  worst-case  complex¬ 
ity  of  these  algorithms  is  0{mn)  where  n  is  the  number  of 
variables  and  m  is  the  size  of  the  discrete  domain  of  each 
variable.  Therefore,  it  is  exponential  in  the  problem  size 
(number  of  variable).  In  contrast,  the  complexity  of  BE  is 
exponential  in  a  parameter  called  the  treewidth  (or  induced 
width )  of  the  problem  and,  if  the  treewidth  is  fixed,  polyno¬ 
mial  in  the  problem  size.  The  induced  width  (Dechter  1999) 
depends  on  the  ordering  of  variables  in  A'.  Therefore,  re¬ 
search  as  been  directed  towards  finding  variable  orderings 
that  minimize  the  treewidth.  Unfortunately,  it  turns  out  to 
be  an  NP-hard  problem;  (Dow  &  Korf  2007)  is  an  example 
of  heuristic  approach.  It  is  important  to  note  that  all  previ¬ 
ous  treewidth  minimization  techniques  can  be  applied  to  the 
continuous  framework  described  in  this  paper  as  well  as  to 
any  other  instance  of  BE. 

BE  has  been  used  successfully  to  solve  Constraint  Opti¬ 
mization  Problems  (Dechter  2003).  The  key  contribution  of 
this  paper  is  to  apply  it  to  scheduling  problems  with  contin¬ 
uous  domains. 


Problem  Formulation 

We  define  a  framework  where  piecewise  linear  value  func¬ 
tions  are  used  to  represent  both  soft  and  hard  constraints  over 
a  set  of  temporal  variables  X  =  {x-i ,  X2,  ■  ■  ■  xn}  with  con¬ 
tinuous  domains. 

Definition  1  For  any  subset  of  continuous  variables  X  = 
{Xix ,  Xi2 , . . .  Xik  },  let  R[X]  denote  the  k-dimensional  con¬ 
tinuous  space  Rfc  where  the  axis  are  labeled  with  variables 
in  X. 

Definition  2  Given  a  subset  of  continuous  variables  X  = 
{xii:Xi2, . .  .xik},  a  (k  +  1  )-tuple  (a0 ,  ah ,  ai2 , .  ..aik)  is 
used  to  represent  several  quantities: 

•  A  linear  constraint  C  that  is  true  in  all  states  x  = 
( Xi j ,  Xi2, . . .  Xik)  such  that  «o  +  5Zz=i  an xk  <  0.  We 
say  that  x  satisfies  C  iff .  C  is  true  in  x; 

•  A  linear  utility  function  u  that  associates  the  re¬ 
ward  u{x)  =  Cto  +  '^2l=laiixii  to  each  state  X  = 
(xik ,  Xi2 , . . .  Xik ).  To  represent  a  hard  constraint,  the  con¬ 
stant  ao  can  take  the  value  — oo; 


•  A  linear  policy  p  implementing  the  control  law  Xi  «o  + 
Xw=i  aiixn  for  a  given  index  i  £  (i\,  *2,  •  •  ■  ik)- 

The  set  X  is  called  the  scope  of  the  constraint,  utility  or 
policy. 

Definition  3  A  convex  polygon  is  defined  as  a  set  of  linear 
constraints,  C  =  {(7,,  «}.  Each  constraint  represents  a  face 
of  the  polygon  ( the  term  face  is  sometimes  used  in  a  two 
dimensional  setting,  and  singular  point  in  ID).  The  points 
belonging  to  the  polygon  are  those  that  satisfy  all  constraints 
in  C.  The  scope  of  a  polygon  is  the  union  of  the  scopes  of  its 
faces. 

Note  this  is  a  loose  definition  of  a  polygon  that  encompasses 
unbounded  pieces. 

Definition  4  A  value  function  piece  is  defined  as  a  pair  P  = 
(C,  u)  where  C  is  a  convex  polygon  and  u  is  a  linear  utility 
function.  To  store  the  optimal  policy,  some  pieces  also  carry 
a  linear  policy  p.  The  scope  of  a  piece  is  the  set  of  variables 
appearing  in  its  edges,  utility  function,  or  policy. 

Definition  5  A  piecewise  linear  value  function  with  scope 
X  =  ,  Xi2 , . . .  Xik  }  is  defined  as  a  collection  of  value 

function  pieces  V  =  {Pj,z}  such  that  the  polygons  of  the 
pieces  Pi  constitute  a  partition  o/R[X],  and  the  scope  of 
each  piece  Pi  is  included  in  X. 

Definition  6  A  linear  constraint  optimization  problem  is  a 
pair  V  =  (X,  V)  where  X  =  {x±,X2,  ■  ■  ■  xn}  is  a  set  of 
continuous  temporal  variables  and  V  is  a  set  of  piecewise 
linear  value  functions  V  =  {Pi,  if  with  scope  included  in 
X.  A  solution  ofV  is  a  state  x  =  (xi . . . ,  xn)  G  R.[X],  A 
solution  x*  is  optimal  if  it  maximizes  5Zv'ev  E  (x). 

Solving  Linear  Constraint  Optimization 
Problems  using  Bucket  Elimination 

Formally,  BE  is  based  on  two  operators  over  functions:  sum¬ 
ming  and  variable  elimination  (Elimj.  The  soundness  of 
BE  for  piecewise  linear  value  functions  is  formally  derived 
from  Theorem  1,  which  we  establish  from  two  Lemmas. 

Lemma  1  The  sum  of  a  finite  number  of  piecewise  linear 
value  functions  is  a  piecewise  linear  value  function. 


loosely  speaking,  to  project  Q  on  XT,).  It  consists  of  com¬ 
puting  a  partition  of  R[X_j]  in  convex  polygons,  as  shown 
in  Fig.  2  for  X  =  {X0lXi}.  This  is  achieved  by  first  of  pro¬ 
jecting  the  pieces  of  Q  on  R[X_j],  which  creates  only  con¬ 
vex  polygonal  pieces,  and  then  computing  all  possible  inter¬ 
sections  of  projected  pieces.  Therefore,  the  partition  com¬ 
puted  by  projection  contains  only  convex  polygonal  pieces. 

Consider  now  a  particular  projected  piece  P°  and  call  Vq 
the  set  of  pieces  of  Q  that  contributes  to  P°  (intersecting 
the  horizontal  stripe  associated  with  If  in  Fig.  2).  If  A0 
as  any  translation  of  the  Xj-axis  that  projects  inside  of  P°, 
then  A0  enters  P  through  a  well  defined  face  Cf  and  ex¬ 
its  through  another  face  Cp  (see  Fig.  2).  Because  utility 
is  linear  over  P,  maximum  utility  is  attained  on  a  face  Cp 
that  is  either  Cf,  or  Cp.  This  allows  deriving  an  analyt¬ 
ical  expression  of  the  optimal  decision  rule  for  X{.  If  the 
optimal  of  piece  P  is  attained  on  the  face  represented  by 

C*P  :  01%  +  Ya= i  af,xji  +  a?xi  <  0  (Cp  =  Cf  or  Cp), 

then  the  optimal  decision  rule  (policy)  for  piece  P  is  fip\ 

Xi  ^  c  nC  xh  ■ 

*  l=i  * 

This  rule  guarantees  that  af  +  afix.'h  +  a? xi  =  0, 
and  thus  that  we  are  choosing  a  point  on  the  face  represented 
by  Cp.  As  long  as  parent  variables  (xj1 ,  Xj2 , . . .  Xjk )  are  in 
P°,  the  optimal  value  of  Q  over  piece  P  is  attained  by  /;  p, 
which  is  a  linear  policy  with  scope  X_j. 

Now  that  the  optimal  policy  for  piece  P  is  known,  it  is 
easy  to  derive  an  analytical  expression  of  the  optimal  value 
of  Q  over  P.  If  up  is  the  linear  utility  associated  with  piece 
P  and  pp  is  the  optimal  policy  in  P,  then  the  optimal  value 
is  the  return  of  u p  when  x,  is  chosen  following  pp.  In  other 

words,  if  up(x)  =  ctg  +^;=i  +aixi  and  up  '■  xi  <— 

ao  +  ]C/'=i  a>j  xji  +  afxj ,  then  an  analytical  expression  of 
Vp  is  obtained  by  substituting  Xi  for  pp  in  up: 

k 

VP(x)  =  a%  +  of  •  o%  +  (a“  +  a"  •  a £ )  xk  .  (3) 

z=i 


Proof.  The  sum  of  two  value  functions  is  computed  by  enu¬ 
merating  all  non-empty  intersections  of  two  of  their  pieces, 
and  summing  the  utility  functions  carried  by  these  pieces. 
Because  the  intersection  of  two  convex  polygons  is  a  con¬ 
vex  polygon,  and  the  sum  of  two  linear  utility  functions  is  a 
linear  utility  function,  it  results  in  a  well  defined  piecewise 
linear  value  function.  □ 

Lemma  2  If  Q  is  a  piecewise  linear  value  function  with 
scope  X  =  [xj1 ,  Xj2 , . . .  Xjk ,  Xj },  then  Elini,(Q)  is 
a  piecewise  linear  value  function  with  scope  X—i  = 
{Xji  !  X02  1  •  •  ■  Xjk  }• 

Proof:  To  prove  this  lemma,  we  show  how  Elim^Q)  can  be 
computed  and  make  clear  that,  by  construction,  the  resulting 
function  is  a  piecewise  linear  value  function. 

The  first  operation  in  the  calculation  of  Elimj(Q)  is  to 
project  the  partition  of  R[X]  defined  by  Q  on  R[X_,]  (or. 


The  resulting  utility  is  linear  with  scope  X_j. 

To  compute  Elim.((Q),  we  need  to  maximize  Q  with  re¬ 
spect  to  Xi.  By  definition: 


V(x°1,4--“4 


)  G  P° 


max 


[Q( 


T°  X° 

xii  ’  xh  ’ 


x°  x 
■xJk,  xz 


max 

Pev° 


[VP(a 


However,  the  piece  P  that  maximizes  Q  may  vary  over  P°. 
Therefore,  P°  may  be  split  in  at  most  as  many  pieces  as  in 
Vq.  For  each  P  G  Vq,  we  build  a  convex  polygon  Cp  by 
first  copying  all  faces  of  I’"  and  then  adding  to  this  set  the 
following  constraints: 

VP'  G  V°Ql  P'  ±  P  :  Up/ (x)  -  uP(x)  <  0  ,  (4) 


Algorithm  1  A  straightforward  implementation  of  Th.  1 
1:  Compute  an  analytical  representation  of  Q  =  XVeB(i) 

2:  Project  Q  on  A'_i. 

3:  for  each  projected  piece  P°  do 

4:  for  each  piece  P  of  Q  that  contributes  to  P°  ( P  G  Vq)  do 

5:  Compute  the  optimal  return  Vp  of  P  following  Eqn.  3. 

6:  Compute  the  constraint  set  Cp  as  in  proof  of  Lemma  2. 

7:  if  Cp  is  consistent  then 

8:  Add  the  piece  {Cp,  Vp)  to  Vt. 

9:  end  if 

10:  end  for 

11:  end  for 


Figure  2:  Projecting  of  Q  on  X_<  =  {xo}:  The  convex  polygonal 
partition  of  the  plane  represents  the  pieces  of  Q.  Horizontal  stripes 
represents  the  partition  of  R[X_i]  obtained  by  projecting  Q. 


where  up  and  up>  are  the  linear  utility  functions  carried  by 
P  and  P' .  These  constraints  are  well  defined  linear  con¬ 
straints  with  scope  X_j.  They  ensure  that  the  optimal  of 
Q  over  P°  is  carried  by  piece  P.  If  the  resulting  set  Cp 
is  consistent,  then  Cp  is  a  well  defined  polygon  and  the 
triple  ( Cp,Vp ,  /. ip )  is  a  well  defined  value  function  piece  of 
Elimi(Q).  Because  Elim?;(Q)  is  made  only  of  such  pieces, 
it  is  a  well  defined  piecewise  linear  value  function.  □ 

Theorem  1  If  bucket  B(i)  contains  only  piecewise  linear 
value  functions,  then  the  function  Vi  resulting  from  an  ap¬ 
plication  of  Eqn.  1  is  also  a  piecewise  linear  value  function. 

Proof:  The  claim  results  directly  from  Lemma  1  and  2.  □ 

Bucket  Elimination’s  fundamental  equation  (Eqn.  1)  starts 
with  a  universal  quantification  over  (xjx ,  Xj2 , . . .  Xjk ) .  In  the 
linear  framework  defined  in  the  previous  section,  these  vari¬ 
ables  have  infinite  continuous  domains.  Therefore,  applying 
Eqn.  1  requires  a  continuous  infinity  of  maximizations.  For¬ 
tunately,  Theorem  1  shows  that  the  computation  can  be  per¬ 
formed  only  once  for  each  piece  of  V, ,  and  there  is  a  finite 
number  of  such  pieces. 

A  direct  application  of  Th.  1  solves 


(iv)  the  maximum  number  of  constraints  in  a  piece  of  a  func¬ 
tion  V  G  B(i).  The  algorithm  is  exponential  in  the  treewidth 
because  it  has  to  build  a  piecewise  linear  value  function  Vi 
with  dimension  as  high  as  the  treewidth. 

While  being  conceptually  simple,  Alg.  1  presents  two  bot¬ 
tlenecks:  (i)  computing  the  multidimensional  sum  ():  (ii) 
projecting  Qi  on  X_,.  Although  it  is  guaranteed  to  be 
polynomial  in  the  problem  size  as  long  as  the  treewidth 
is  bounded,  these  operations  become  expensive  when  the 
treewidth  increases.  This  is  because  the  size  of  the  largest 
function  Vi  that  the  algorithm  has  to  build  is  equal  to  the 
induced  width  of  the  problem.  For  instance,  calculating 
Q  requires  computing  all  non-empty  intersections  of  pieces 
of  functions  in  B(i).  Computing  non-empty  intersections 
is  done  by  considering  all  possible  combinations  of  pieces, 
putting  all  the  faces  of  these  pieces  in  a  single  set,  and  testing 
the  consistency  of  this  set.  If  the  set  is  consistent,  then  the 
pieces  have  a  non-empty  intersection.  In  high  dimensions, 
this  consistency  test  is  expensive,  making  the  summation  an 
overly  expensive  process.  In  the  next  section,  we  present  an 
algorithm  that  avoids  these  two  pitfalls. 

A  Witness  Algorithm 

Algorithm  2  is  a  witness  algorithm  that  efficiently  imple¬ 
ments  Th.  I.1  This  algorithm  associates  with  every  piece  P 
of  Vi  a  witness  state-vector  xp  =  {xPt ,  xP2 , . . .  xPk )  that  tes¬ 
tify  of  the  existence  of  P  in  V, .  It  then  performs  the  finite 
calculation 
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this  computation  being  performed  analytically.  That  is,  an 
analytical  expression  of  Lj  (and  the  associated  optimal  pol¬ 
icy)  that  applies  to  the  whole  piece  P  is  computed  instead 
of  a  single  numerical  value.  In  this  way,  Eqn.  1  is  implicitly 
solved  over  the  infinite  set  of  values. 

Algorithm  1  is  a  straightforward  procedure  computing 
Eqn.  5.  It  follows  closely  the  proof  of  Lemma  1  and  2.  It 
is  important  to  note  that,  for  a  fixed  treewidth,  the  complex¬ 
ity  of  this  procedure  is  polynomial  in  the  size  of  the  problem, 
which  is  a  vector  four  variables:  (i)  The  number  of  temporal 
variables  n;  (ii)  The  number  of  value  functions  V  G  Pit)', 
(iii)  the  maximum  number  of  pieces  in  a  function  V  G  B(i)\ 


The  result  of  this  local  computation  is  generalized  to  pro¬ 
duce  analytical  expressions  of  the  optimal  utility  and  pol¬ 
icy  over  P.  The  strength  of  this  algorithm  is  that  once 
{xjx  ,Xj2,...  Xjk )  has  been  substituted  by  (xj^ ,  xP2 , . . .  xPk ), 
then  all  functions  in  P(i)  are  piecewise  linear  functions  of 
the  single  variable  Xj,  that  is  V {xf}  =  a  +  axXi.  Therefore, 
summing  them  and  maximizing  them  is  extremely  easy. 

:We  borrow  the  term  witness  from  the  POMDP  literature. 
Littman’s  witness  algorithm  uses  witness  points  to  testify  for  the 
presence  of  an  “cc-vectors”  in  the  optimal  solution  of  a  POMDP 
(Kaelbling,  Littman.  &  Cassandra  1998). 


Algorithm  2  A  witness  algorithm 

1:  while  Vi  is  not  totally  defined  do 

2:  Pick  a  state  x°  =  ( x ,  x°2 .  x°k )  £  R[X_;]  where  Vi  is 

not  defined  yet. 

3:  Determine  the  characteristics  of  the  piece  of  Vi  containing 

x°  (utility  u  and  policy  p). 

4:  Build  a  set  of  constraints  ensuring  that  these  characteristics 

remain.  This  constraint  set  represent  the  faces  of  the  piece 
of  Vi  containing  x°. 

5:  Simplify  the  constraint  set  built  at  previous  step. 

6:  Add  the  new  piece  to  V . 

7:  end  while 


This  contrast  strongly  with  Alg.  1  that  endures  very  high 
cost  manipulating  multidimensional  functions. 

Selecting  an  unexplored  state  x°  €  K[X_j]  (line  2  of 
Alg.  2):  This  step  is  a  bottleneck  where  we  face  the  mul¬ 
tidimensional  space  avoided  otherwise.  It  can  be  performed 
systematically  by  a  costly  and  complex  recursive  procedure. 
Random  and  pseudo-random  techniques  might  be  more  ef¬ 
ficient  despite  having  no  guarantee  of  completeness.  Our 
prototype  implementation  employs  a  simple  but  inefficient 
technique  that  checks  only  the  points  on  a  discrete  grid  cov¬ 
ering  the  set  of  possible  values  for  the  variables  in  X 

In  the  following  we  call  Py  the  piece  of  Vt  containing  x°. 
x°  is  the  witness  of  Py.  The  next  steps  aim  at  building  Py  . 

Determining  the  characteristics  (//  and  u)  of  Py  (line  3 
of  Alg.  2)):  Our  algorithm  performs  these  operations  in  a 
particular  way  that  minimizes  computation  time. 

Given  x°  =  (xj± ,  x®2 , . . .  Xjk ) ,  the  first  operation  per¬ 
formed  is  to  substitute  (xjL ,  Xj2 , . . .  Xjk )  for  x°  in  every 
function  V  £  B(i).  The  resulting  function  called  V[x°] 
is  a  piecewise  linear  function  of  the  single  variable  x, : 
V[x°](xi)  =  cto  +  OiiXi .  We  later  call  this  operation  in¬ 
stantiating  V  as  x°. 

Next  all  V[x°]  are  summed  to  produce  the  function  Q[x°] 
that  is  the  instantiation  of  Q  =  V  'n  x°-  I*1  Fig-  2, 

Q[x°)  represents  the  variation  of  Q  along  the  A0  axis  that 
passes  through  x°.  Again,  these  operations  are  straightfor¬ 
ward  and  the  result  is  a  linear  function  of  xt  only. 

Finally  we  determine  the  maximum  of  Qfx0]  w.r.t.  Xi. 
This  is  a  straightforward  optimization  of  a  1 -dimensional 
piecewise  linear  function.  We  then  have  solved  BE  funda¬ 
mental  equation  in  the  particular  point  x°  with  minimum 
effort.  We  obtain  an  analytical  expression  of  the  optimal 
utility  u  and  the  optimal  policy  p  in  Py  by  generalizing  this 
local  computation  as  explained  below. 

Because  the  V[x°]’s  and  Q[x°]  are  unidimensional  func¬ 
tions,  their  faces  are  singular  points  between  two  adja¬ 
cent  pieces.  One  can  show  that  each  singular  point  of 
V[x°]’s  and  Q[x°]  is  inherited  from  an  face  of  some  func¬ 
tion  V  £  B{i).  First,  V[x°]  is  obtained  by  traversing  the 
piecewise  linear  value  function  V  with  the  axis  A0  defined 
by  (xj1 ,  Xj2 , . . .  Xjk )  =  x°,  as  in  Fig.  1 .  The  singular  point 
of  V[x°]  are  the  points  where  A0  intersects  an  face  of  V. 


Therefore  each  singular  point  is  inherited  from  a  constraint 
C  of  V.  Second,  Q[x°]  is  obtained  by  intersecting  and  sum¬ 
ming  the  unidimensional  piecewise  linear  functions  V[x0]. 
Therefore,  each  of  its  singular  point  is  inherited  from  a  sin¬ 
gular  point  of  some  function  V[x°],  and  so  it  is  inherited 
from  a  constraint  C  of  V.  In  order  to  determine  the  analyti¬ 
cal  solution,  we  must  keep  track  of  the  relationships  between 
singular  points  of  Q[x°]  and  constraints  of  V’s;  fortunately, 
maintaining  this  information  is  relatively  inexpensive. 

Since  Q[x°]  is  piecewise  linear,  its  optimum  is  attained 
in  one  of  its  singular  points.  The  constraint  at  the  origin  of 
this  singular  point  determines  the  optimal  policy  over  Py.  If 
the  constraint  is  C  :  1  aflXjl  +  af  Xi  <  0  then 

the  optimal  policy  of  Vp  is  given  by  Eqn.  2.  This  is  a  well 
defined  linear  policy  with  scope  in  „Y_j. 

Now  that  we  know  the  policy  /i  attached  to  Vp,  we  want  to 
determine  an  analytical  expression  of  the  utility  u  attached 
to  this  piece  of  V, .  We  cannot  carry  on  the  same  computa¬ 
tion  as  in  the  proof  of  Lemma  2  because  we  do  not  have  an 
explicit  representation  of  Q.  However,  again,  keeping  track 
of  simple  information  achieves  the  same  result.  Indeed  we 
only  need  to  memorize  the  piece  of  each  V  that  contributes 
to  each  piece  of  Q[x°],  which  is  straightforward  and  very 
cheap.  Then,  if  we  denote  P*  as  the  piece  of  Q[x°]  con¬ 
taining  the  optimal  value,  we  can  trace  back  the  piece  Py 
of  each  V  £  B{i)  that  contributes  to  P*.  By  summing  the 
utility  functions  carried  by  these  pieces,  we  get  an  analytical 
expression  of  the  utility  of  the  piece  of  Q  containing  P*.  As 
in  the  proof  of  Lemma  2,  we  know  the  utility  and  the  optimal 
policy  associated  with  the  optimal  piece  of  Q.  Therefore,  we 
can  get  an  analytical  expression  of  the  utility  attached  to  Vp 
by  substituting  x%  by  p  in  u  following  Eqn.  3. 

The  preceding  discussion  shows  that  there  is  no  need  to 
employ  a  complex  multidimensional  representation  of  Q  to 
determine  the  optimal  policy  and  utility  associated  with  Vp. 
In  this  calculation,  the  most  costly  step  is  the  instantiation  of 
each  function  V  in  x°.  We  show  below  how  this  operation 
can  be  performed  efficiently. 


Instantiating  V  £  B(i)  in  x°  =  (xjk ,  Xj2, . . .  x®k)  (used 
at  line  3  of  Alg.  2):  The  procedure  used  for  instantiating 
a  function  V  in  x°  loops  through  all  pieces  of  V  and,  for 
each  piece  P  =  (C,  u),  checks  whether  the  A°-axis  passing 
through  x°  traverses  P  or  not.  This  check  is  performed  by 
substituting  (xj1 ,  Xj2 , . . .  Xjk )  by  x°  in  every  constraint  C  £ 

C.  It  transforms  a  constraint  C  :  a o  +  Xw=i  akxH  —  0  into 
unidimensional  constraint  C[x°]  that  is  is  either 
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Figure  3:  Instantiating  V :  A0  (the  axis  defined  by 
( Xj1 ,  Xj2, . . .  Xjk)  =  x°)  enters  P  through  face  C~  =  C4  and 
leaves  through  face  C+  =  Ci.  For  each  constraint  C,  K(C)  is  the 
coordinate  (on  A0  )  of  the  point  where  A0  traverses  face  C. 


We  call  C+  the  set  of  constraints  of  the  first  type  and  C  the 
set  of  constraints  of  the  second  type.  Then  we  define 
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as  the  coordinate  (on  A0)  of  the  point  where  A0  traverses 
face  C  (see  Fig.  3),  and  C+  =  arg  mincec+  [K(C)\ 
and  C~  =  argmax^gc-  \K(C)\.  Then  A0  traverses  V 
iff.  K(C~)  <  K(C+).  If  A0  traverses  V  then  the  segment 
of  A0  contained  into  P  defines  a  piece  [ K{C~ ),  K(C+ )]  of 
V[x°] .  The  constraint  associated  with  singular  point  I\  (C~ ) 
is  C~  and  the  constraint  associated  with  K(C+)  C+ .  The 
detail  of  this  computation  is  presented  in  Alg.  3. 


Computing  the  faces  of  the  new  piece  (line  4  of  Alg.  2): 

In  the  proof  of  Lemma  2,  the  faces  of  a  new  piece  are  com¬ 
puted  by  adding  to  the  faces  of  P°  (see  Fig.  2)  the  constraints 
defined  by  Eqn.  4.  We  follow  the  same  general  principle  to 
derive  the  set  of  faces  of  Pf . 

First,  because  we  do  not  have  an  explicit  representation 
of  Q,  we  do  not  have  an  easy  access  to  the  faces  of  P°. 
Fortunately,  an  equivalent  constraint  set  can  be  recovered 
using  a  simple  set  of  pointers.  The  constraints  of  a  piece 
P°  of  the  projection  of  Q  on  A_,  express  the  fact  that  the 
axis  A0  passing  through  x  =  {xj1 ,  Xj2 , . . .  Xjk )  traverses  the 
same  sequence  of  pieces  of  Q  for  all  x  in  P°  (cf.  Fig.  2).  A 
necessary  condition  for  this  holding  is  that  A0  traverses  the 
same  sequence  of  pieces  of  each  V  £  B(i).  Given  a  function 
V  £  B(i ),  we  can  build  a  set  of  constraints  implying  that  A0 
traverses  a  fixed  sequence  of  pieces  of  V  in  two  steps: 

•  For  each  piece  P  of  V  that  is  traversed  by  A0,  add  a  con¬ 
straint  implying  that  P  is  still  traversed  by  A0; 

•  For  each  piece  P  of  V  that  is  not  traversed  by  A°,  add  a 
constraint  implying  that  P  is  still  not  traversed  by  A0. 

This  procedure  is  linear  in  the  number  of  pieces  in  V.  Since 
it  is  performed  once  for  each  piece  P°,  the  algorithm  is 
quadratic  in  the  maximum  number  of  pieces  in  a  value  func¬ 
tion.  Therefore,  we  adopt  the  cheaper  alternative  that  builds 
a  single  set  of  constraints: 


Algorithm  3  Instantiating  V  in  x°  =  (x^ ,  x°2 , . . .  x®k ) . 
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for  each  piece  V  =  (C,  u)  of  V  do 

Cf  <-  0,  C+  <-  0. 

for  each  face  C  =  (ao,  a ,  aj2 , . . .  ajk ,  on)  of  C  do 
if  Xi  is  in  the  scope  of  C  ( cm  ^  0)  then 
if  an  >  0  then 
C+  ^C+  U{C}. 

else 

C~  ^-C~U{C}. 

end  if 

Compute  =  ■ 

else 

//  Xi  is  not  in  the  scope  of  C  (cti  =0) 

Substitute  (xj± ,  Xj2 , . . .  Xjk )  for  x°  in  C  and  check  for 
consistency:  ao  +  ^i=i  a^x0^  <  0  ? 
if  Consistent  then 

Skip  to  next  face  of  C  (go  to  line  3). 
else 

Skip  to  next  piece  of  V  (go  to  line  1). 

end  if 
end  if 
end  for 

Compute  C~  =  arg  maxCgC-  [K(C)]  and  C+  = 
argminC6C+  [K(C)). 
if  K(C+)  <=  K(C~)  then 
//Inconsistency  detected 
Skip  to  next  piece  of  V  (go  to  line  1). 
else 

Substitute  (xj1 ,  Xj2 , . . .  Xjk )  for  x°  in  u  to  get  a  function 

u[x°](xi). 

Add  the  piece  (\K(C~),K{C+)],u[x°\)  to  l/[a:0].  At¬ 
tach  to  singular  point  K(C~ )  the  constraint  C~  and  at¬ 
tach  C+  to  K(C+). 

end  if 

//  Continue  to  next  piece  of  V 

end  for 

Return  V[x°] 


•  For  each  piece  P  of  V  that  is  traversed  by  A0,  add  a  con¬ 
straint  implying  that  A0  still  enters  and  exits  P  through 
the  same  faces. 

This  creates  an  equivalent  set  of  constraints,  but  the  com¬ 
plexity  is  linear  in  the  number  of  pieces  traversed  by  A0 
which  is  generally  much  smaller  than  the  total  number  of 
pieces.  Now,  given  a  piece  P  of  V,  we  ensure  A0  enters 
and  exit  through  the  same  face  of  P  using  two  types  of  con¬ 
straint: 


VC  €  C~,  C  ±  C-  :  K(C)  <  K(C~)  , 


and 


VC  £C+,C  ^C+  :  K(C+)  <  K(C)  . 

We  add  a  following  constraint  to  ensure  A0  traverses  P: 

K(C~)  <  K(C+)  . 


Formally,  if  C 


*0 


Ek 
1=1 


ctj,  Xi,  <  0,  and  C+  :  af  + 


A-l  u ii  u'*( 

Ef=!  aXxH  -  and  C  =  a0  +  E;fc=i  aHxH  <  0,  then 

these  constraints  constraints  are  computed  as  (respectively): 
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proof  of  Lemma  2,  this  piece  is  split  in  as  many  sub-pieces  as 
there  are  pieces  P  of  Q  that  satisfy  Eqn.  4  for  some  x.  Here, 
we  are  not  interested  in  all  these  pieces  but  only  the  one  con¬ 
taining  the  current  witness  x°.  Thus,  we  complete  the  set  of 
faces  of  Py  by  adding  the  constraints  defined  by  Eqn.  4  for 
the  particular  piece  P  that  carries  the  optimum  of  of  Q[x°] 

as  computed  at  step  1.  If  up(x)  =  cto  +  J2j=i  a:ixji  and 
up'{x)  =  a'0  +  J2kj= i  a'jixji'  Eqn.  4  becomes 

k 

a'0  -  a0  +  (a'-,  -  ajt)  xjt  <  0  .  (11) 

1=1 


Equations  7  to  1 1  generate  the  set  of  all  faces  of  Py. 


Figure  4:  The  partition  of  R2  defined  by  the  pieces  of  Q.  In  this 
example,  there  are  two  value  functions  in  Bp.  A  reward  function 
Ri(xi)  at  the  origin  of  the  horizontal  faces  in  the  figure,  and  a 
function  f?oi  (®i  —  *o)  that  represents  piecewise  linear  preferences 
on  the  difference  between  xo  and  x\  and  that  induces  the  diago¬ 
nal  constraints.  This  is  a  standard  situation  encountered  in  many 
STPPs.  The  axis  A0  and  A1  traverse  the  same  sequence  of  pieces 
of  both  R\  and  f?oi  •  However,  they  traverse  different  pieces  of  Q. 
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Equations  7  to  9  allow  for  the  building  of  a  constraint 
set  that  guarantees  that  A°  traverses  the  a  fixed  sequence 
of  pieces  of  each  V  £  B{i).  This  condition  is  necessary  to 
ensure  that  we  stay  inside  of  a  single  piece  P°,  but  it  is  not 
sufficient.  A  counter-example  is  provided  in  Fig.  4.  Fortu¬ 
nately,  this  issue  can  be  addressed  by  adding  a  small  number 
of  simple  constraints.  In  the  example  of  Fig.  4,  A0  and  A1 
traverse  the  same  sequence  of  pieces  of  each  V  £  B{i), 
but  they  traverse  different  sequences  of  pieces  all  functions 
taken  together.  Both  axis  start  in  the  first  piece  of  R\  and 
the  first  piece  of  f?oi,  A°  enters  the  second  piece  of  If  be¬ 
fore  entering  the  second  piece  of  i?oi,  while  A1  does  the 
opposite.  Therefore,  we  get  a  complete  set  of  constraints  to 
characterize  the  faces  of  P°  in  the  following  way:  For  any 
two  adjacent  pieces  [a,  6]  and  [b,  c]  of  Q[x°],  such  that  Ca  is 
the  constraint  associated  to  singular  point  a  during  the  con¬ 
struction  of  and  Cb  is  the  constraint  associated  with  b, 
we  add  the  constraint: 


K{Ca)  <  K(Cb)  . 


If  Ca  :  ag  +  Ya=i  xh  <  0  and  Cb\ab0  +  Ya=i  au  xu  < 

0,  then  it  translates  as: 
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(10) 


The  constraints  defined  by  Eqn.  7  to  10  are  the  faces  defin¬ 
ing  a  unique  piece  P°  of  the  projection  of  Q  on  X_j.  In  the 


Simplifying  Constraint  Sets  (line  5  of  Alg.  2):  The  con¬ 
straint  set  built  by  the  procedure  described  above  may  con¬ 
tain  redundant  constraints,  or  constraints  that  are  useless  be¬ 
cause  they  are  dominated  by  other  constraints.  For  instance, 
if  C  :  Xq  <  0  is  in  the  constraint  set,  then  any  constraint 
C'  :  Xo  <  K  for  some  K  >  0  can  be  omitted  in  the  rep¬ 
resentation  of  the  piece.  Indeed,  our  experiments  show  that 
the  witness  algorithm  produces  many  such  constraints. 

The  algorithm  complexity  is  sensitive  to  the  number  of 
constraints  in  the  description  of  each  piece.  Although  the 
maximum  number  of  constraints  in  a  piece  is  guaranteed  to 
be  polynomial  in  the  problem  size,  it  is  preferable  to  keep 
this  number  as  low  as  possible.  Therefore,  in  line  5  of  Alg.  2 
we  try  to  purge  the  set  from  useless  constraint. 

Simplifying  a  set  C  of  linear  constraints  with  scope  X  is 
a  difficult  problem.  At  this  point  we  are  considering  several 
approaches  to  address  it: 

•  A  similar  problem  arises  in  the  optimization  of  POMDPs 
where  we  must  determine  a  minimal  set  of  linear  faces  to 
represent  a  piecewise  linear  convex  function.  Here  the  prob¬ 
lem  is  addressed  by  solving  multiple  LPs.  This  approach 
seems  relatively  easily  transferable  to  our  sub-problem. 

•  Solving  the  dual  program  of  an  LP  M  allows  determining 
the  constraints  that  are  binding  in  the  optimal  solution  of  M 
(Schrijver  1986).  Defining  M  =  (A,  V,  0)  where  0  is  the 
objective  function  that  returns  0  in  all  states,  then  any  point 
inside  of  the  polygon  defined  by  C  is  an  optimal  solution 
of  M.  Solving  the  dual  problem  of  M  can  thus  compute  a 
minimum  set  of  faces. 

•  Our  current  implementation  of  the  witness  algorithm  just 
prunes  the  set  C  from  unary  constraints  that  are  dominated 
by  other  unary  constraints.  Useless  fc-ary  constraints,  k  >  1 
are  kept  in  the  description  of  the  piece,  which  can  hurt  the 
algorithm.  In  general,  there  is  a  trade-off  between  the  time 
spent  pruning  the  set  of  constraints,  and  the  impact  of  useless 
constraints  on  complexity. 

Note  that  the  problem  of  simplifying  C  is  exponential  in  the 
number  of  variables  in  its  scope  X.  However,  in  our  case, 
the  scope  of  the  functions  is  bounded  by  the  treewidth  of  the 
problem.  Therefore,  with  a  bounded  treewidth,  complexity 
is  polynomial  in  the  size  of  the  problem. 


[0,  20] 


Figure  5:  A  simple  STPP  with  non-convex  preferences.  Interval 
[h,  Ui]  attached  to  vertex  ti  represents  the  hard  constraint  L  <ti  < 
Ui.  Interval  [Uj ,  Uij]  on  edge  ti  — ►  tj  represents  the  hard  constraint 
lij  <  tj  —ti<  Uij.  There  is  a  total  of  4  value  functions  in  B( 2): 
one  for  the  hard  constraint  on  t2,  two  for  the  hard  constraints  on 
£2  —  to  and  £2  —  £1,  and  the  reward  R2(t-2)  represented  at  the  top. 

Summary  and  Future  Work 

This  paper  provides  proof  of  concept  that  bucket  elimina¬ 
tion  can  be  used  to  optimize  a  rich  model  of  temporal  pref¬ 
erences.  The  potential  of  this  concept  for  scheduling  with 
preferences  is  twofold.  First,  a  richer  preference  model 
opens  up  generation  of  schedules  that  better  reflect  the  de¬ 
sires  of  the  modeler;  second,  the  alternate  position  of  BE 
in  the  space-time  trade-off  opens  up  a  complementary  ap¬ 
proach  compared  to  prior  art. 

After  showing  the  relevance  and  soundness  of  the  ap¬ 
proach,  we  provided  a  grounded  algorithm.  A  prototype  im¬ 
plementation  is  capable  of  solving  a  richer  class  of  problems 
than  previous  approaches.  It  has  been  run  successfully  on 
problems  involving  non  convex  and  fc-ary  constraints.  Fig.  6 
shows  the  result  of  one  iteration  of  the  witness  algorithm  on 
the  STPP  of  Fig.  5. 

At  this  point,  our  implementation  of  the  witness  algorithm 
suffers  from  two  weaknesses:  (i)  the  simplistic  grid-base 
procedure  used  to  find  a  new  witness  x°  where  Vi  is  not  yet 
defined;  (ii)  the  procedure  used  to  simplify  constraint  sets 
cannot  handle  more  than  unary  constraints. 

Our  ongoing  work  is  to  improve  the  efficiency  of  the  pro¬ 
totype  implementation,  and  to  compare  the  BE  approach 
both  theoretically  and  empirically  to  alternate  approaches, 
such  as  the  use  of  repeated  Linear  Programming  with  a 
branch-and-bound  search. 

In  particular,  because  binary  and  convex  STPPs  is  a  sub¬ 
class  of  problems  that  can  be  represented  as  linear  constraint 
optimization  problems,  it  would  be  interesting  to  compare 
our  algorithm  with  the  results  in  (Morris  et  al.  2004).  BE 
and  LP  approaches  can  both  solve  this  sub-class  of  prob¬ 
lems,  but  there  complexity  bears  on  different  parameters. 
Our  future  work  will  aim  at  showing  experimentally  that  BE 
can  be  a  competitive  approach  for  large  problems  with  small 
treewidth. 
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Figure  6:  The  3D  value-function  V2  obtained  by  eliminating  £2  in 
the  problem  of  Fig.  5.  Points  not  assigned  a  value  in  this  graph  are 
those  that  break  a  hard  constraint  (hence  they  have  —00  value). 
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